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Abstract 

Large-scale data centers and cloud computing have turned 
system configuration into a challenging problem. Several 
widely-publicized outages have been blamed not on soft¬ 
ware bugs, but on configuration bugs. To cope, thousands of 
organizations use system configuration languages to manage 
their computing infrastructure. Of these. Puppet is the most 
widely used with thousands of paying customers and many 
more open-source users. The heart of Puppet is a domain- 
specific language that describes the state of a system. Pup¬ 
pet already performs some basic static checks, but they only 
prevent a narrow range of errors. Furthermore, testing is in¬ 
effective because many errors are only triggered under spe¬ 
cific machine states that are difficult to predict and repro¬ 
duce. With several examples, we show that a key problem 
with Puppet is that configurations can be non-deterministic. 

This paper presents Rehearsal, a verification tool for Pup¬ 
pet configurations. Rehearsal implements a sound, complete, 
and scalable determinacy analysis for Puppet. To develop it, 
we (1) present a formal semantics for Puppet, (2) use sev¬ 
eral analyses to shrink our models to a tractable size, and 
(3) frame determinism-checking as decidable formulas for 
an SMT solver. Rehearsal then leverages the determinacy 
analysis to check other important properties, such as idem- 
potency. Finally, we apply Rehearsal to several real-world 
Puppet configurations. 

Categories and Subject Descriptors F.3.1 [Logics and 
Meanings of Programs]: Specifying and Verifying and Rea¬ 
soning about Programs—Mechanical verification 

Keywords Puppet, system configuration, domain-specific 
languages, verification. 


1. Introduction 

Consider the role of a system administrator at any organi¬ 
zation, from a large company to a small computer science 
department. Their job is to maintain computing infrastruc¬ 
ture for everyone else. When a new software system, such as 
a Web service, needs to be deployed, it is their job to pro¬ 
vision new servers, configure the firewall, and ensure that 
data is automatically backed up. If the Web service receives 
a sudden spike in traffic, they must quickly deploy additional 
machines to handle the load. When a security vulnerability 
is disclosed, they must patch and restart machines if neces¬ 
sary. All these tasks require the administrator to write and 
maintain system configurations. 

Not too long ago, it was feasible to manage systems by 
directly running installers, editing configuration files, etc. 
A skilled administrator could even write shell scripts to 
automate some of these tasks. However, the scale of modern 
data centers and cloud computing environments has made 
these old approaches brittle and ineffective. 

System configuration languages. System configuration is 
a problem that naturally lends itself to domain-specific lan¬ 
guages (DSLs). In fact, the programming languages com¬ 
munity has developed several DSLs for specifying sys¬ 
tem configurations that are used in practice. For exam¬ 
ple, Nixos Go) uses a lazy, functional language to de¬ 
scribe packages and system configurations; Augeas Q uses 
lenses to update configuration files; and Engage HD 
provides a declarative DSL that tackles issues such as inter¬ 
machine dependencies. 

In the past few years, several system configuration lan¬ 
guages have also been developed in industry. Puppet, Chef, 
and Ansible (recently acquired by Red Hat) are three promi¬ 
nent examples. This paper focuses on Puppet, which is the 
most popular of these languages, but these commercial lan¬ 
guages have several features in common that set them apart 
from prior research. First, they support a variety of oper¬ 
ating systems, tools, and techniques that systems adminis¬ 
trators already know. Unlike NixOS, they don’t posit new 
package managers or new Linux distributions, but simply 
use tools like apt and rpm under the hood. Second, these 
languages provide abstractions for managing several kinds 
of resources, such as packages, configuration files, user ac- 
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counts, and more. Therefore, they are broader in scope than 
Augeas, which only edits configuration files. Finally, these 
languages provide relatively low-level abstractions, com¬ 
pared to earlier work like LCFG m. For example. Puppet 
provides a simple and expressive DSLs that encourages av¬ 
erage users to build their own abstractions. 

Puppet. Puppet configurations (called manifests) are writ¬ 
ten in a expressive, yet constrained DSL, which makes them 
amenable to analysis. To a first approximation, a manifest 
specifies a collection of resources, their desired state, and 
their inter-dependencies. For example, the following Pup¬ 
pet manifest states that the vim package should be installed, 
that the user account carol should exist, and that she should 
have a . vimrc file in her home directory containing the sin¬ 
gle line syntax on: 

package{^viin’: ensure => present } 

file{Vhome/carol/.vimrc’: content => ^syntax on’ } 

user{’carol’: ensure => present; managehome => true } 

It’s tedious to describe every individual resource in this man¬ 
ner, so Puppet makes it easy to write parameterized modules. 
The official module repository. Puppet Forge, has nearly four 
thousand modules from over six hundred contributors. 

Non-determinism and modularity. A key property of Pup¬ 
pet is that manifests should be deterministic j^ . Determin¬ 
ism is a critical property because it helps ensure that a man¬ 
ifest has the same effect in testing and in production. Simi¬ 
larly, if one manifest is applied to several machines, which is 
common in large deployments, determinism helps to ensure 
that they are replicas of each other. 

Unfortunately, it is easy to write manifests that are not de¬ 
terministic. Puppet can install resources in any order, unless 
the manifest explicitly states inter-resource dependencies]^ 
Therefore, the example manifest above is non-deterministic: 
there will be a runtime error if Puppet tries to create the file 
/home/carol/.vimrc before Carol’s account. We can fix 
this bug by making the dependency explicit: 

User[’carol’] -> File[’/home/carol/.vimrc’] 

The fundamental problem is that Puppet manifests spec¬ 
ify a partial-order on resources, thus resources can be in¬ 
stalled in several orders. However, when some dependencies 
are missing, applying the manifest can go wrong: the system 
may signal an error or may even fail silently by transitioning 
to an unexpected state. These bugs are very hard to detect 
with testing, since the number of valid permutations of re¬ 
sources becomes intractable very quickly. 

Surprisingly, a manifest can also have too many depen¬ 
dencies and be over-constrained. Imagine two manifests A 
and B that both install the resources i?i and i? 2 . Suppose 
that Ri and R 2 do not depend on each other, but the manifest 
authors take a conservative approach and add a false depen- 

' Puppet calculates dependencies automatically only in some trivial cases, 
e.g., files “auto-require” their parent directory. 


dency to avoid non-determinism issues. If A picks i?i -> i ?2 
and B picks i ?2 -> Ri then A and B cannot be composed. 

Therefore, manifests must be deterministic to be correct, 
but must only have essential dependencies to be compos- 
able. Without composability, manifests cannot be decom¬ 
posed into reusable modules, which is one of the key fea¬ 
tures of Puppet. However, when a manifest is only partially- 
ordered, we may need to check an intractably large number 
of orderings to verify determinism. 

A further complication is that the Puppet has a diverse 
collection of resource types, which makes it hard to deter¬ 
mine how resources interact with each other. For example, a 
file may overwrite another file created by a package, a user 
account may need the /home directory to be present, a run¬ 
ning service may need a package to be installed, and so on. 
We could try to side-step this issue by building a dynamic 
determinacy analysis HEl. However, a purely dynamic 
approach could only identify a problems when two replicas 
diverge, whereas a static determinacy analysis helps ensure 
that a manifest behaves correctly on any machine regardless 
of its initial state. 

Idempotency. Determinism is not a sufficient condition to 
ensure that Puppet behaves predictably. In a typical deploy¬ 
ment, the Puppet background process periodically reapplies 
the manifest to ensure that the machine state is consistent 
with it. For example, if a user modifies the machine {e.g., 
manually editing configurations), re-applying the manifest 
will correct the discrepancy. Thus if the machine state has 
not changed, reapplying the manifest should have no effect. 
Like determinacy, this form of idempotence is also believed 
to be a key property of Puppet m . However, it is also trivial 
to construct manifests that are not idempotent. 

Our approach. To the best of our knowledge, this is the 
first paper to develop programming language techniques for 
Puppet (or a related language such as Chef and Ansible). We 
first present a core fragment of Puppet with several small 
examples that illustrate its problems (section]^. We develop 
a formal semantics of Puppet that models manifests as pro¬ 
grams in a simple, non-deterministic imperative language of 
filesystem operations called FS (section]^. 

Our main technical result is a sound, complete, and scal¬ 
able determinacy analysis (section]^. To scale to real-world 
examples, we use three different analyses to shrink the size 
of models. The first two analyses dramatically reduce the 
number of paths that the determinism-checker needs to rea¬ 
son about by eliminating resources that do not affect deter¬ 
minism and eliminating other side-effects that are not ob¬ 
served by the rest of the program. The third analysis is an 
unusual commutativity check that accounts for the fact that 
resources are mostly idempotent. Finally after leveraging the 
aforementioned analyses, the determinacy checker encodes 
the semantics as effectively-propositional formulas for an 
SMT solver. 
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Types rtype ::= file | package | • • • 


Strings 

str ::= "• • •$x- -Sy- • •" 


Identifiers 

3^ $x 1 $y 1 ■ ■ ■ 


Titles 

t ::= str \ x 


Values 

V ::= str 

String 


1 

Number 


1 [r>l ■ ■ ■ Vn\ 

Array 


X 

Variable 

Attributes 

attr \\= str => V 


Resources 

R\:= rtype{t\ attri ■ ■ ■ attr-a} 


Manifests 

m ::= R 

Resource 


1 def±TLertype(xi---Xn) 

{ m } Type 


1 rtype-^^ [fi] -> rtype 2 U2'] 

Dependency 


1 mi m2 

Composition 


Figure 1: Syntax of Puppet fragment used in this paper. 


define myuser($title) { 
user {"$title": 

ensure => present, 
mcuiagehome => true 

} 

file {"/home/${title}/.vimrc": 
content => "syntax on" 

} 

User ["$title"] -> File["/home/${title}/.vimrc"] 

} 

myuser {"alice": } 
myuser {"carol": } 

Figure 2: A user-defined resource type and its instantiations. 

We argue that our determinacy analysis enables several 
other higher-level properties to be checked (section |^, and 
show this is the case by developing a simple idempotence 
checker that leverages determinism in a fundamental way. 

We implement our algorithms in a tool called Rehearsal, 
which we evaluate on several real-world examples (sec¬ 
tion]^. Finally, we discuss related work (section]^, sum¬ 
marize the limitations of our approach (section]^, and con¬ 
clude (section]^. The Rehearsal source code, benchmarking 
scripts, and a technical appendix are available online llT3l . 

2. Introduction to Puppet 

This section introduces the fragment of Puppet that we 
use in the exposition of this paper. We also illustrate the 
kinds of problems that Rehearsal solves. 

2.1 A Core Fragment of Puppet 

The Puppet DSL is quite sophisticated. It has typical fea¬ 
tures such as functions, loops, and conditionals, and sev¬ 
eral domain-specific features that make it easy to specify 
resources and their relationships. Rehearsal can parse and 
process a significant subset of Puppet, but, for clarity, we 
constrain our examples to the fragment of Puppet shown 
in figure [T] A manifest, m, is composed of resources, re¬ 
source type declarations, and inter-resource dependencies. 
A resource, R, has a type, a title, and a map of attributes. 


The resource type determines how the attribute-map is inter¬ 
preted. For example, a file resource must have an attribute 
called path, a user resource must have an attribute called 
name, and so on. The resource title can be any descriptive 
string, but is often used as the default value for an essen¬ 
tial attribute. For example, if a file resource does not have 
the required path attribute, the title is used as the path. 
A manifest can declare several resources by juxtaposition, 
but the order in which resources appear is not significant. 
Instead, manifests must specify dependencies explicitly. To 
state that the resource t 2 depends on the resource ti, we 
write rtype^ [fi] -> rtype 2 [^ 2 ! gin addition to a few dozen 
built-in resource types. Puppet allows manifests to define 
their own types. A type definition is essentially a function 
that consumes named attributes as arguments and produces 
a manifest as a result. For example, if all users in an orga¬ 
nization use the same default environment, we can create a 
new type called myuser and instantiate it for several users, 
as shown in figure 

2.2 Common Puppet Problems 

There are a number of problems that can easily occur in 
Puppet manifests. 

Non-deterministic errors. A common Puppet idiom is to 
first install a package and then overwrite its default config¬ 
uration. For example, the apache2 package installs a web 
server and several configuration files. To host a website, at 
least the default site configuration file, 000-default. conf, 
has to be replaced (figure [3^. If the dependency between the 
package and the file is accidentally omitted. Puppet may try 
to create the configuration file first which would signal an 
error because the file is in a directory that the package has 
yet to create. 

Over-constrained dependencies. Consider a strawman so¬ 
lution to the non-determinism problem: we could add false 
dependencies so that all resources are totally ordered. Unfor¬ 
tunately, this approach makes it difficult to write indepen¬ 
dent modules which is one of the main features of Puppet. 
For example, figure shows two simple types that config¬ 
ure C-H- and OCaml development environments^Both mod¬ 
ules install make and m4 because they are commonly used by 
C-H- and OCaml projects. To force determinism, both mod¬ 
ules in the figure have false dependencies between make and 
m4. However, each has picked a different order which can 
easily occur when the modules have different authors. There¬ 
fore, if we try to instantiate both modules simultaneously. 
Puppet will fail and report a dependency cycle. ^This heavy- 
handed approach to determinism sacrifices composability. 

^ The first letter of a type name is capitalized in resource references. 

^ Idiomatic Puppet would use the class keyword. 

^Readers familiar with Puppet may know that shared resources have to 
be guarded with defined. Some people consider defined to be an anti¬ 
pattern, but a simple search shows that it is used in over l/3rd of the 
packages on Puppet Forge to enable the kind of modularity that we discuss. 
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file {"/etc/apache2/sites-available/OOO-default.conf": 
content => ..., 

} 

package{"apache2": ensure => present } 

(a) Signals an error nondeterministically. 
define cppO { 

package{’in4’: ensure => present } 
package{'make’: ensure => present } 
package{’gcc’: ensure => present } 

Package[’m4’] -> Package['make’] 

Package['make'] -> Package['gcc'] 

} 

define ocamlO { 

package{'make': ensure => present } 
package{'m4': ensure => present } 
package{'ocaml': ensure => present } 

Package['make'] -> Package['m4'] 

Package['m4'] -> Package['ocaml'] 

} 

(b) Cannot be composed due to false dependencies. 

package{'golang-go': ensure => present } 
packagef'perl': ensure => absent } 

(c) Leads to two different success states. 

filef'/dst": source => "/src" } 
filef'/src": ensure => absent } 

File["/dst"] -> File["/src"] 

(d) Not idempotent. 

Figure 3; Several problematic manifests. 


Silent failure. In addition to non-deterministic errors, it is 
also possible to write a manifest that nondeterministically 
leads to two distinct states without Puppet reporting an er¬ 
ror. For example, the manifest in figure states that Perl 
should be removed and the Go compiler should be installed. 
Surprisingly, on Ubuntu 14.04, the Go compiler depends on 
Peri ls , so this state is not realizable. Puppet cannot de¬ 
tect this problem, but simply dispatches to the native pack¬ 
age manager {e.g., apt or yum) to actually install and re¬ 
move packages. For this manifest. Puppet issues two low- 
level commands to remove Perl and install Go. Since there 
are no dependencies, they may execute in either order. If Perl 
is first removed, the command to install Go installs Perl too, 
but if Perl is removed after Go is installed, that command 
will remove Go too. This kind of error is more insidious than 
a nondeterministic error, since there isn’t an obvious fix. 

Non-idempotence. Another key property of Puppet mani¬ 
fests is that they should be idempotent; applying a manifest 
twice should be the same as applying it once. However, Pup¬ 
pet does not enforce this property, which makes it easy to 
produce manifests that are not idempotent. For example, we 


Vertices V ::= tti | • • • | uj. 

Edges B C V X V 

Vertex Labels L £ V ^ R 

Resource Graphs G S V X E X L 

Figure 4: Resource graphs. 

can make the non-deterministic manifest in figure deter¬ 
ministic by removing Perl before Go is installed; 

Package['perl'] -> Package['golang-go'] 

However, this manifest is not idempotent. Puppet checks 
which packages are installed before it issues any commands 
to install or remove packages. In this example, if both pack¬ 
ages are already installed. Puppet will remove Perl and take 
no further action, even though removing Perl removes Go. If 
we apply the manifest again (i.e., when neither package is in¬ 
stalled), Puppet installs Go and takes no further action, even 
though Perl is implicitly installed. The real issue is that this 
manifest is fundamentally inconsistent and cannot be fixed 
by adding dependencies. A system cannot have Perl removed 
and Go installed, so the manifest should be rejected. 

An even simpler example of non-idempotence is the man¬ 
ifest in figure]^ which copies src to dst and then deletes 
src. The second run of this manifest will always fail, be¬ 
cause the first run removes src. This example shows that 
even though primitive resources are designed to be idempo¬ 
tent, they can be composed in ways that break idempotence. 

Summary. We’ve introduced a small fragment of Puppet 
and used it to illustrate several kinds of bugs that can occur 
in Puppet manifests. We’ve argued that the root cause of 
these bugs is that Puppet does not ensure that manifests 
are deterministic and idempotent. Before we describe how 
Rehearsal checks these properties, we present the semantics 
of Puppet that Rehearsal uses. 

3. Semantics of Puppet 

This section presents a semantics for Puppet, which we 
develop in two stages. (1) We compile manifests to a di¬ 
rected acyclic graph of primitive resources, which we call 
a resource graph. The compilation process involves sev¬ 
eral passes to eliminate features that inject dependencies, 
change attributes, and so on. We also substitute instantia¬ 
tions of user-defined types with their constituent resources 
until only primitive resources remain. (2) Next, we model 
the semantics of individual resources as programs in a small 
imperative language of file system operations called FS. We 
carefully design FS so that it is expressive enough to de¬ 
scribe the semantics of resources, yet restrictive enough to 
enable the static analyses we present in subsequent sections. 

3.1 From Puppet to Resource Graphs 

A resource graph G is a directed acyclic graph with vertices 
labeled by primitive resources. An edge exists from Vi to Vi 
if Vi depends on Li. At a high-level, we compile manifests 
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into resource graphs by converting primitive resources to 
nodes and dependencies to edges. To do this, we employ a 
number of passes to simplify manifests. 

Puppet has several abstractions that allow manifests to 
succinctly describe dependencies. For example, user-defined 
types can be used to abstract over a collection of other re¬ 
sources and dependencies. We reduce user-defined abstrac¬ 
tions to their constituent resources by repeatedly substituting 
their definitions until only primitive resources remain. In or¬ 
der to preserve ordering, this pass must introduce new edges 
between resources within instances of abstractions. In ad¬ 
dition, resources can also be assigned to a stage, and stages 
are ordered independently of resources. To deal with this, we 
implement a stage elimination pass that adds edges between 
the constituent resources of each stage. 

Certain Puppet features have non-local side effects. For 
example, the following expression uses a resource collector 
to update all file-resources owned by carol to be unreadable 
by others, regardless of where they are defined; 

File<| owner == ^carol’ |> { mode => "go-rwx" } 

Unfortunately, resource collectors are not modular and make 
separate compilation impossible. In general, it is not possi¬ 
ble to know the attributes of a resource until all user-defined 
types (which may define collectors) are eliminated as de¬ 
scribed above. The passes that tackle these kinds of expres¬ 
sions are necessarily global transformations. 

Our compiler tackles the details described above and 
some other features of Puppet that we don’t belabor here. 

3.2 From Resources to FS Programs 

Puppet has dozens of different primitive resource types that 
can interact with each other in subtle ways. Moreover, some 
resources have flags that dramatically change their behavior. 
To deal with this diversity, we model resources as small pro¬ 
grams in a low-level language called FS that captures their 
essential effects and possible interactions. The advantage of 
using FS is that we can quickly add support for additional 
resource types and new versions of Puppet without rebuild¬ 
ing the rest of our analysis toolchain. In this paper, FS is an 
imperative language with simple operations that affect the 
filesystem. However, it also would be straightforward to en¬ 
rich the language in several ways. 

Syntax and semantics of FS. The FS language, defined 
in figure l^is a simple imperative language of programs that 
manipulate the filesystem. We model filesystems (cr) as maps 
from paths (p) to file contents. A file may be a regular file 
with some content (File(sfr)) or the value Dir that repre¬ 
sents a directory. Expressions in FS denote functions that 
consume filesystems and produce either a new filesystem 
or error (err). FS has primitive expressions to create direc¬ 
tories (mkdir(p)), create files (creat(p, sfr)), remove files 
and empty directories (rm(p)), and copy files (cp(pi,p2))- 
Sequencing (ei;e2) and conditionals (if (a) ei else 62) be¬ 
have in the usual way. Predicates include the usual boolean 


connectives and primitive tests to check if a path is a 
file (file?(p)), a directory (dir?(p)), an empty directory 
(emptydir?(p)), or contains nothing (none?(p)). 

Since FS has no loops, its programs always terminate. 
This is a reasonable restriction since applying Puppet re¬ 
sources must terminate too. FS does not have procedures 
or variables, but their omission doesn’t affect programmers, 
since FS code is generated from a host language (in our 
case, Scala). A more important restriction is that FS pro¬ 
grams work with a finite set of paths and file contents, so 
FS programs are finite state. At first glance, it appears that a 
program would not be affected by the state of paths that do 
not appear in the program text. However, the semantics of 
rm(p) and emptydir?(p) are affected by subpaths of p, even 
if they don’t appear in the program. Finally, FS programs 
only work with a finite set of file contents. In fact, there are 
no operations that allow programs to read the contents of 
files, but this is not an essential property. 

FS can easily be extended in several ways to produce 
higher-fidelity models of Puppet resources, e.g., it is easy 
to imagine adding timestamps, file-permissions, and so on. 
Notably, these extensions would not affect the finiteness of 
FS programs, so we believe our analysis approach would 
work with these higher-fidelity models too. 

Notation. We write ci = 62 when both expressions pro¬ 
duce the same output (or error) for all input filesystems. For 
brevity, we use if (ei) 62 as shorthand for if (ei) 62 else id. 

3.3 Modeling Resources as FS Programs 

Now that we have a language of filesystem operations, we 
define a compilation function C : i? —> e that maps resources 
to FS expressions. The actual definition has several hundred 
lines of code and is quite involved, but the high-level idea is 
to model each resource as an FS program. Even for simple 
resources, C needs to validate attributes, fill-in values for 
optional attributes, and produce programs that check several 
preconditions before applying the desired action. We now 
illustrate how C models several key resource-types. 

Files and directories. Individual files and directories are 
the simplest resource in Puppet. The file resource type 
manages both and has several attributes that determine ( 1 ) 
whether it is a file or directory, ( 2 ) if it should be created 
or deleted, (3) if parent directories should be created, (4) 
the contents of a file, or (5) a source file that is copied 
over. Moreover, all combinations of these attributes are not 
meaningful, and most are optional. The C function addresses 
these details in full. 

SSHkeys. Some Puppet resources edit the contents of con¬ 
figuration files. Eor example, the ssh_authorized_key 
resource manages a user’s public keys, where each re¬ 
source is an individual line of a single file. Rather than 
increase the complexity of FS by including detailed file¬ 
editing commands, we model the logical structure of these 
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Syntax 



Semantics 

Paths 

p::= / 

Root directory 

[< 


1 p/str 

Sub-path 

[file?(p)l 

File Contents 

V ::= Dir 

Directory 

[dir?(p)] 


1 File(5?r) 

File 

[none?(p)] 

File Systems 

a ::= {pi=vi--- 

II 

[emptydir?(p)] 

Predicates 

a ::= none?(p) 

Does not exist 



1 file?(p) 

Is a file 

[ 


dir?(p) 

Is a directory 

Nl 


1 emptydir?(p^ 

Is an empty dir. 

[err] 


1 true 

True 


1 false 

False 

|[mkdir(p/str)] 


1 ai V a2 

Disjunction 

|[creat(p/str, 


1 ai A a2 

Conjunction 



Negation 

[i-mlp)] 

Expressions 

e ::= id 

No op 


1 err 

Halt with error 



1 mkdir(p) 

Create directory 

[cp(pi,p2/str)l 


1 creat(p, str) 

Create file 



rm(p) 

Remove file/empty dir. 

[ei;e2l 


1 cpiPl,P2) 

Copy file 


1 ei;e2 

Sequencing 

[if (a) ei else 62] 


1 if (a) ei else 

62 Conditional 


I G cr ^ bool 

• = 3str.cr{p) = File(5?r) 

- A a{p) = Dir 

• = p 0 dom{a) 

■ = cr(p) = Dir and —>3str.p/str G dom(a) 

I G cr ^ (7 + err 

A 

■ = cr 

A 

• = err 

f a[p/str: = Dir] [dir?(p) A none?(p/str)] cr 

I err othemise 

I (j[p/str: = File(5'?r')] [dir?(p) A none?(p/str)] cr 
I err otherwise 

(cr —p [file?(p) V emptydir?(p)] fj 
I err otherwise 

^ o-\p 2 /str: = f\\e{str')] |[none?(p 2 /str) A dir?(p 2 )] cr 
and(7(pi) = File(sir') 

I err otherwise 

\le 2 \a' [ei]o- = o-' 

I err [ei] cr = err 

l[eilo- 

l[ 62 lcr Otherwise 


Figure 5; FS syntax and semantics. 


resources in a portion of the filesystem disjoint from other 
files. However, this alone disguises a certain kind of de- 
terminacy bug. Consider a manifest with two resources: an 
ssh_authorized_key and a file that overwrites the key- 
file. Clearly, these resources do not commute, but by plac¬ 
ing ssh keys in their own disjoint directory, the compiled 
program would be deterministic. To address this issue, our 
model for ssh_authorized_key also creates a key-file and 
sets its content to a unique value, enabling us to catch this 
kind of determinacy bug. 

Packages. A package resource creates (or removes) a large 
number of files and directories, so we need this file list to 
model packages. Fortunately, there are simple command¬ 
line tools that do exactly this: e.g., apt-file for Debian- 
based systems, repoquery for Red Hat-based systems, and 
pkgutil for Mac OS x|^The C function invokes the afore¬ 
mentioned tool and builds a (potentially very large) program 
that first creates the directory tree and then issues a sequence 
of creat(p, str) commands to create the files. In our model, 
we simply give every file p in a package a unique content str. 
This model is sound but conservative: some equivalences can 
be lost. For example, suppose a manifest has two resources: 
a package that creates a file p and a file resource that over¬ 
writes p with exactly the same contents as the package. This 
manifest would be deterministic without any dependencies, 
but our tool would report it as nondeterministic, due to our 
conservative package model. However, this situation is un¬ 
likely to arise in practice, but if it does it may indicate an¬ 
other mistake: it’s more likely that the author meant to over¬ 
write p with some other contents. 

^ We’ve tested with apt-f ile and repoquery. 


[G1 e cr -)■ 2 '^+^" 

[G] a = {[C(L(ui)); ■ ■ ■ ;C(L(r„))] cr | (i;i ■ ■ ■ r„} e perms(G)} 
where G = {V, E, L) 

Figure 6: Semantics of resource graphs. 


Other resource types. We model several other resource 
types, including cron jobs, users, groups, services, and host- 
file entries. Puppet has several resources types that are only 
applicable to Mac OS X or Windows systems that we have 
not modeled. However, if we wished to analyze a manifest 
for these platforms, it should be easy to extend the resource 
compiler to support these resources. Notably, the rest of our 
toolchain would be unchanged as it is agnostic to the actual 
set of resources since it operates over FS programs. 

3.4 Semantics of Resource Graphs 

Now that we have a compiler from resources to FS, it is 
straightforward to give a semantics to resource graphs. Infor¬ 
mally, a resource graph denotes a function from filesystems 
to a set of filesystems and the error state. To define this func¬ 
tion, we take all sequences of resources that respect the order 
imposed by the edges, compile each resource-sequence to a 
sequence of FS programs, apply each program to the input 
state, and take the union of the results (figure]^. 

A pleasant feature of this definition is that the resource 
graph and resource compiler abstract away the peculiarities 
of Puppet. We can extend C to support new resource types 
or the Puppet compiler to support even more Puppet features 
without changing the methods that will be discussed in the 
rest of this paper. 
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4. Determinacy Analysis 

This section presents our main technical result which is a 
sound, complete, and scalable approach to check that re¬ 
source graphs (produced from manifests) are deterministic. 

Definition 1 (Determinism). A resource graph G is deter¬ 
ministic, if for all filesystems a, ||G] a\ = 1. 

This property does not preclude a manifest from always 
producing an error on some or even all inputs. Any non¬ 
trivial manifest makes assumptions about the initial state 
{e.g., the operating system in use) and thus will raise an error 
if it is applied to a machine that is not in the right initial state. 
Determinism simply guarantees that successes and failures 
will be predictable. 

Our approach has three major steps: 

1. The first step is to reduce the number of paths that we 
need to reason about. Even a small manifest may ma¬ 
nipulate several hundred paths and tracking their state 
over hundreds of intermediate states can be intractable. 
We observe that resources often modify paths p that are 
not accessed by any other resource, thus operations on 
these paths can be safely eliminated without affecting the 
result of the determinism-check. 

2. The next step is to reduce the number of permutations 
of the resource graph, which can grow exponentially 
with the number of resources. The natural approach is 
to use partial-order reduction with a fast, commutativity 
check. However, the obvious approach, based on calcu¬ 
lating read- and write-sets is not effective because many 
resources may create overlapping directories {e.g., /usr 
and /etc). We observe that this is a form of false shar¬ 
ing and develop a commutativity check that accounts for 
idempotent directory creation. 

3. The final step is to encode the semantics of the manifest 
as a decidable formula for an SMT solver that is satisfi- 
able if and only if the program is non-deterministic. Our 
encoding relies on the fact that programs manipulate a fi¬ 
nite set of paths that are statically known. However, the 
result of some operations may be affected by the state of 
paths that do not appear in the program itself. We care¬ 
fully bound the domain of paths to ensure our approach 
is complete. 

We first present our encoding of manifests as formulas. 

4.1 From Resource Graphs to Formulas 

The function <i)(e) produces a collection of formulas that en¬ 
code the semantics of the expression e (figure 0 . In these 
formulas, two boolean variables determine whether the ini¬ 
tial and final states are non-error states and every path is 
modeled by two variables that describe their initial and fi¬ 
nal state. These path-state variables are only meaningful in 
a non-error state. More concretely, a logical state (E) is a 
record of two components: ( 1 ) E.ok is a formula that is true 


dom(a) S 2 ^ 
dom(f\\e? (p)) = {p} 

(iom{emptydir?(p)) = {p,p/str} str is fresh 
dom{e) S 2*’ 

dom(mkdir(p/sfr)) = {p,p/str} 
dom{cre 3 t{p/str, str')) = {p/str,p} 

dom (rm(p)) = {p/str} str is fresh 
dom(cp(pi, p2/str)) = {pi, p2, p2/str} 

dom{ei; 62) = dom(ei) U dom{e2) 
dom(if (a) ei else 62) = dom(a) U dom{e\) U dom{e2) 

Figure 8 : Bounding the domain of FS programs. 

if the current state is not the error state and (2) S.fs maps 
paths to formulas that describe their state. We could employ 
McCarthy’s theory of arrays lfT9l to encode this map, but 
it’s more efficient to encode it directly with one formula per 
path. To encode resource graphs G as formulas, we use the 
function ^a{G), defined in the same figure, which maps the 
input logical state to a set of output logical states by evalu¬ 
ating each expression on the fringe with $(e) and recurring 
on the subgraph that has e removed. 

To prove this encoding sound and complete, we need 
to relate concrete states returned by the evaluator to logi¬ 
cal states. This is mostly routine, but the domain of logi¬ 
cal filesystems has to be large enough: if a program reads 
or writes to a path p, then there must be a formula p G 
dom(S.fs). For example, note that mkdir(/a/b) reads /a 
and writes /a/b. 

Lemma 1 (Soundness and completeness). For all a and G: 

1. o' G |G]| o iff there exists E G 4>g({o/c = true, fs=o),e) such 
thatT, h {o/c = true, fs=o'}. 

2. err G |G]] a iff there exists E G 'l>G({o/c = true, fs-o},e) such 
that E.o/c h false. 

4.2 Checking Determinism 

With resource graphs encoded as formulas, it should be 
straightforward to use a theorem prover to check determin¬ 
ism (though we have yet to address scalability issues). Since 
$g(G) maps an input logical state to a set of output logical 
states, the resource graph should be deterministic, if and and 
only if there does not exist an input logical state that pro¬ 
duces two different logical states, i.e., the following formula 
should be unsatisfiable: 

3Ei, Yi2, S 3 .S 2 G $g(G)Ei a E 3 G $( 3 (G)Ei a E 2 7 ^ E 3 
The subtlety here is that the domain of ^^(G) to be 
large enough to find a counterexample when G is a non- 
deterministic resource graph. 

To understand the issue, consider the simpler problem of 
checking whether two expressions are inequivalent, ei ^ 62, 
which is the essence of checking non-determinism. At first 
glance, it appears that expressions only read and write to the 
paths that appear in it and the result of an expression is not 
affected by the state of any other paths. That is, if we have 
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Logical Formulas (/)::=■■■ 

Logical Filesystems a ::= (pi = = (pj.) 

Logical States E ::= (ok = ^, fs = (t) 

encPred(d-, b) £ (p 

ok(e) £ a bool 
ok{\d)d- = true 
ok{err)d- = false 

ok{mkd\r{p/str))d- = < 5 ’(p) = dir A &{p/str) = dne 
ok{creat{p/str^ str'))d- = d-{p/str) = dne A d-{p) = dir 
o/c(rm(p))(T = 3 c.(j(p) = file(c)A 

Vstr.p/str £ dom{&) => &{p/str) = dne 
ok{cp{pi,p2/str))& ^ 3 str' .d-{pi) = File(str')A 

&{p2) = Dir A a{p2/str) = none? 
ok(ei; 62)0- ^ ok{ei)d- A ok{e2){f{ei)a) 
ok{\f (b) ei else 62)0^ — if {encPred[a^ b)) ok{ei)& else ok{e2)d- 


fie) £&->■& 

/(id)CT = CT 
/(err)o- = CT 

/(mkdir(p/str))CT = d\p/str:= Dir] 
f{creat{p/str, str'))d = a{p/str ■.-i\\e{str')] 

/(rm(p))CT = a{p : = none?] 

/(cp(pi, p2/str))a- = a\p2/str : = d-(pi)] 

/(ei;e 2 )d- = /(e 2 )(/(ei)( 5 -) 

/(if ( 6 ) ei else 62)0' = if {encPred{a, b)) f{ei)a else f(e2)d- 
$(e) e S -)■ S 

<I>(e)(ok = b, fs = ( 3 ') = (ok = b A ok{e)a, fs = /(e)(7) 

$G(G)eS^ 2 ^ 

4 -G(( 0 ,i?))S = {S} 

$G(G)S^U'I>G(G-e)($(e)S) 
where inDegreeie) = 0 


Figure 7: Encoding FS as logical formulas. 


a state a such that |ei] tr ^ |e 2 ] a then for paths p that do 
not appear in either ei or 62, (|ei] <j){p) = (|e2] cr){p). But, 
this equation is wrong. 

The emptydir?(p) predicate poses a problem, since it de¬ 
pends on the state of the immediate children of p, includ¬ 
ing those that may not appear in the program. Consider the 
following inequality, where the only difference between the 
programs is that one checks if the directory is empty and the 
other only checks that it is a directory: 

if (emptydir?(/a)) id else err 
^ if (dir?(/a)) id else err 

Any input filesystem that demonstrates the inequality must 
have a file (or directory) within /a. However, if we construct 
a logical filesystem using only the paths that appear in the 
program text, we will not find this counterexample. A similar 
problem affects rm(p). The function in figure addresses 
this problem by adding fresh files in directories that are 
removed or tested for emptiness to avoid this bug. We can 
now prove that equivalence-checking is complete. 

Lemma 2 (Completeness—equivalence). If: 

• |ei] tr f |e2| cr and 

• dom[a') = dom{ei) U dom{e2) 

then $((o/( = true, fs- a'), ei) f $((ok= true, fs- ct'), 62 ). 

Soundness is straightforward. A model for the formula 
can be easily transformed into a counterexample filesystem. 

Lemma 3 (Soundness—equivalence). If 

• $((o/( = true, fs- a),ei) f $((o/( = true, fs- ct), 62 ) and 

• (7 h (7 

then |ei] cr f |e2] cr. 

We use these lemmas to prove that that determinism 
checking is sound and complete. In the theorem below, 
dom{e) is lifted to dom{G) in the obvious way. 

Theorem 1. (Determinism) G is deterministic, if and only if 
there exists Ei, E2, and'S^ such that 'E12 S $g'(G')EiAE3 S 


$g(G)Ei a E2 E3 is unsatisfiable, where dom(Ei) = 
dom{^2) = domlYif) = dom{G). 

4.3 Commutativity and Directory Creation 

Modeling all valid permutations of resources can produce 
formulas that are intractably large. For example, suppose a 
resource graph G has exactly two nodes a and b that do not 
have any ancestors. The naive approach considers evaluat¬ 
ing either node first and then recurs on the two subgraphs 
G — a and G — b. When the sub-graphs also have several 
nodes without any ancestors, the size of the generated for¬ 
mula grows intractably large. A significantly better approach 
is to use a fast, syntactic commutativity check to rule out per¬ 
mutations that don’t need to be explored, similar to partial- 
order reduction. Note that it is not sufficient to check that 
a and b commute. For example, b; a; c and b; c; a are valid 
permutations in the following graph: 

a b -> c 

We can only conclude that they are equivalent if we know 
that a commutes with both b and c. Therefore, to avoid 
recurring on both G — a and G — b, we need to prove that a 
(or b) commute with all nodes that are not ancestors of a, as 
shown in figure [9a| 

Next, we need a fast, syntactic commutativity check, 
which should be straightforward to do for FS. Surprisingly, 
the natural approach does not work. A typical commutativity 
check works as follows: to check if ei and 62 commute, cal¬ 
culate the set of locations that each reads and writes. If the 
expressions don’t have any overlapping writes and ei does 
not read any locations that 62 writes (and vice versa), then 
they do commute. If not, they may or may not commute and 
we need to semantically check both orderings. 

This approach is not effective for Puppet, due to the se¬ 
mantics of packages. Typical packages install files to shared 
directories (e.g. /usr/bin, /etc, and so on) and will cre¬ 
ate these directories if necessary. Therefore, the conventional 
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$G(G')eS^-2^ 

#G(G)S^-I>G(G-e)($(e)S) 

where inDegree{e) = 0 

Ve' S G.^ancestor(e', e) => e'; e = e; e' 
$G(G)S^U«-G(G-e)(#(e)S) 
where inDegree{e) = 0 

(a) Incorporating the commutativity-check. 


Abstract Values v L \ R\W \ D 
Abstract State a ::= {pi=vi ■ ■ ■ pk = v^.) 

±nR,DnW 

le]c & a ^ a 

( o'[p/str: = D] d-{p/str)^D 
[if {—>d\r?{p/str)) mkd\r{p/str)]cd- ^ < andd-{p) = D 

I a \p/str : = W] otherwise 
[mkdir(p)]c'(T ^ o-[p: = W] 

[creat(p, str)]cd' = a[p:--W] 

[ei; e2]G5' = [e2]G([ei]c'r) 


(b) Checking commutativity. 

Figure 9: Commutativity checks eliminate the number of permutations that need to be generated. 


approach cannot prove that packages commute. Manifests 
that installs several packages typically do not specify any 
dependencies between them, so this issue arises frequently. 

To address this issue, we use an abstract interpretation 
that maps each path p to the abstract values _L, R, W, and D 
(figure [9b]i. These values indicate that the expression either 
does not affect p (_L), reads from p (R), writes to p (W), or 
ensures that p is a directory (D). A mkdir(p) expression that 
doesn’t first check if p already exists is simply a write (W). 
Only a guarded mkdir(p) can ensure p is a directory, such as 
these expressions: 

if (^dir?(p)) mkdir(p) 

= if (none?(p)) mkdir(p) else if (file?(p)) err else id 
In addition, the analysis ensures that expressions create di¬ 
rectory trees in a reasonable order. For example, an expres¬ 
sion that creates /a before /a/b is not equivalent to an ex¬ 
pression that tries to create /a/b before /a. However, two 
expressions that create sibling directories do commute. To 
ensure that these properties hold, we map p/str to D, only 
if p is already mapped to D. 

We can use the result of this analysis to check that ex¬ 
pressions commute, even if they create overlapping directory 
trees. 

Lemma 4. For all ei and 62 , if: 

1. {p I [ei]G-L(p) = i?} n {p I [e2]c-L(p) = VF} = 0 , 

2. {p I [ei]G-L(p) = IF} n {p I [e 2 ]c-L(p) = i?} = 0, 

3. {p I [ei]G-L(p) = D} n {p I [e2]c-L(p) € {R, W}} = 0 , and 
4 - {P I [ei]G-L(p) G {R, W}} n {p I [e2]G-L(p) = D} = 0 

then ei; 62 = 62 ; Ci. 

4.4 Pruning Files from Resources 

The syntactic commutativity check mentioned above elimi¬ 
nates the need to explore different permutations of resources 
that are obviously equivalent to each other. However, even a 
single permutation that installs several large resources can 
make formulas needlessly large. For example, suppose a 
manifest installs a large package (e.g., git, which has over 
500 files) and then doesn’t read or write to any of the files 


that the package creates. Intuitively, we should be able to 
completely eliminate resources that are not observed by the 
rest of the manifest. 

However, there are situations where resources must inter¬ 
fere. It is quite common for a manifest to update a default 
configuration file created by a package. For example, the 
manifest in figurej^installs the Apache web server and sup¬ 
plies a site-specific configuration file that should overwrite 
the default configuration. Even in this situation, the manifest 
does not update most of the other 200 h- files that the Apache 
package creates. Intuitively, we should be able to shrink re¬ 
sources so that we don’t have to track the state of files that 
cannot affect the outcome of the determinism-check. 

In this section, we formalize these two observations using 
two simple analyses. 

Eliminating Resources. Notice that a determinism-check 
is essentially a conjunction of equivalence-checks between 
all valid permutations of resources. For example, the follow¬ 
ing resource graph has eight valid permutations of the four 
resources shown: 

a -> c <- b d 

A naive determinism-check would generate all permutations 
and verify that they are equivalent: 

a',b;c;d = a;b;d;c = a-,d-,b;c = b;a;c;d 
= b-,a;d;c = b-,d-,a;c = d;a-,b;c = d-, b; a; c 

However, suppose we use our commutativity check to de¬ 
termine that c and d commute. We could then rewrite all 
the permutations that end with c; d to instead end with d; c, 
which gives us a series of permutations that all end in c. In 
general, ei; e = 62 ; e, if and only if ei = 62 . Therefore, we 
can completely eliminate c without changing the result of the 
equivalence check. 

In general, if a resource commutes with all other re¬ 
sources that may be evaluated after it in the resource graph, 
then that resource can be eliminated without affecting the 
result of the determinism-check. Moreover, eliminating one 
resource often allows their parents to be eliminated. For ex¬ 
ample, suppose that b commutes with a and d. Eliminating 
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c, as discussed above, allows us to then eliminate b by the 
same argument. However, trying to eliminate b hrst would 
fail, since it does not commute with c, which may-succeed 
b. In practice, a true dependency a ^ b indicates that b truly 
depends on the effects of a and thus the two resources do 
not commute. In our experience, we’ve found it most effec¬ 
tive to eliminate resources by starting with resources at the 
fringe of the dependency graph that are not required by any 
other resources. 

Shrinking Resources. There are several cases where large 
resources cannot be completely eliminated. However, they 
can be shrunk as follows. In general, if a resource writes to 
a path p such that ( 1 ) other resources in the manifest do not 
observe the state of p and ( 2 ) other resources in the manifest 
do not affect the state of p then we can eliminate writes to 
p without changing whether the manifest is deterministic or 
not. Moreover, the encoding of FS programs as formulas can 
then exploit the fact that p is read-only and use a single 
variable to represent the state of p, instead of using new 
variables for each state. This can dramatically reduce the 
number of variables needed to encode the program. 

Consider the problem of shrinking two expressions ei and 
62 to and 62 , such that ei = 62 if and only if = 62 . If 
both expressions leave a path p in the same state, it should 
be possible to shrink both expressions by removing their 
writes to p. However, to implement idempotent operations, 
resources tend to have a complex series of reads and writes 
(section [33] i. Nevertheless, a resource that writes to p typi¬ 
cally ensures that p is either placed in a dehnite state or sig¬ 
nals an error if it cannot do so. We say that these resources 
make definitive writes to p. Therefore, if both expressions 
make the same dehnitive write to p, then we can eliminate 
writes to p. 

We detect dehnitive writes using the abstract interpreta¬ 
tion sketched in hgure fTOb] which produces an abstract heap, 
a that maps paths p to abstract values that characterize the 
effect of an expression on p over all input states; 

• If (t(p) = dir, the expression ensures that p is a directory 
(or signals an error). 

• If a{p) = file(sfr), the expression ensures that p is a hie 
with contents str{oi signals an error). 

• If ( 7 (p) = dne, the expression ensures that p does not 
exist (or signals an error). 

• If a{p) = _L, the expression does not read or write p. 

• If o'(p) = T, the expression has an indeterminate effect 
on p. 

Lemma 5. If (leil.)(p) \Z T then for all states and a 2 , 
del (ai))(p) = ([el (a 2 ))(p). 

If the abstract interpretation determines that ei and 62 set 
a path p to the same dehnite value, we should be able to 
prune writes to p from both expressions. However, consider 


the two equivalent expressions below; 
mkdir(p/sfr); if (dir?(p/sfr)) id else err = mkdir(p/sfr) 
The expressions on either side ensure that p/ sir is a direc¬ 
tory. However, if we naively replace mkdir(p/sfr) with id, 
we get the following wrong result; 

id; if (dir?(p/sfr)) id else err ^ id 
To correctly eliminate writes to p, we need to also transform 
expressions that read from p to account for the effect that the 
write would have had. In our example, the test dirl(p/str) 
will always be true, since it follows mkdir(p/sfr). The in¬ 
sight is that when writes to p are eliminated, we need to 
transform all expressions that subsequently read or write to 
p. In our example, we need to transform dir?(p/sfr) to true. 

The pruning function, prune{p, e), eliminates writes to p 
by preserving reads in this manner (hgure [T 0 a| . The function 
correctly handles programs where a write to p is followed by 
other reads and writes to p by partial evaluation. 

The following lemma states that the same dehnitive write 
from 61 and 62 doesn’t change their (in-)equivalence. 

Lemma 6 . Tf (|ei]_L)(p) = (|e2]-L)(p) = v and v C T 
then 61 = 62 if and only if prune{p, 61 ) = prune(p, 62 ). 

Although pruning eliminates writes to p, it does not elim¬ 
inate reads from p. However, eliminating writes ensures that 
p is a read-only path. When we encode the expression as 
a logical formula, the encoding can optimize for read-only 
paths by using a single variable to represent the initial state 
of the path, which then remains unchanged. 

Pruning for determinism checking. Since a determinism 
check encodes equalities between all permutations of re¬ 
sources, we could also apply the abstract interpretation to 
all permutations, but this would be intractable. Instead, we 
apply the abstract interpretation to each resource in isola¬ 
tion to hnd paths that are dehnitively written by exactly one 
resource and only prune these paths. This conservative ap¬ 
proach works well in practice. 

4.5 Summary 

These are the three major techniques that Rehearsal uses 
to make determinism-checking scale. We’ve also outlined 
how each step preserves (in-)equivalences, so the approach 
is sound and complete. 

Other approaches. We have tried two other techniques 
for checking determinism that are less effective than the 
methodology discussed in this section. 

1. We developed a dynamic analysis that simply installed 
resources in different valid permutations within indepen¬ 
dent Docker containers. The Docker API makes it easy to 
see how a container has updated its hlesystem. However, 
installing resources takes time and it took our prototype 
several hours to verify small manifests with less than ten 
resources. (We fully utilized a four-core machine with 16 
GB RAM.) In contrast, our static analysis checks deter¬ 
minism in seconds. 
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Abs. Values v ::= ± | T | dir | file(5tr) | dne 
Abs. State & ::= (pi = vi ■ ■ ■ =Vk) 


P|[-] Eexpxcr^excr 
P[id] p <J = (id, a) 

P|[err] p cr = (err, cr) 

P|[mkdir(p)] p (t = (err, cr) if cr(p) = Dir or cr{p) = File(5fr) 

P|[mkdir(p/sfr)] p/str cr = (if (none?(p/sfr) A dir?(p)) id else err, cr[p/sfr: = Dir]) 
P[mkdir(p^)] p <j = (mkdir(p^), <j) ifp ^ p' 

P[creat(p, sir)] p cr = (err, cr) if cr(p) = Dir or cr(p) = File(sir') 

P|[creat(p/sir, str')1 p/str a = (if (none?(p/sfr) A dir?(p)) id else err, cr[p/sfr: = File(s£r')]) 
P|[creat(p', sir)] p cr = (creat(p', sir), cr) if p p' 

P[rm(p)| p (T = (err, cr) if p 0 dom(cr) or 3str.p/str E dom{a) 

P|[rm(p)| p cr = (if (file?(p) V emptydir?(p)) id else err, a — p) 

P[rm(p')] pa = ^m(p'). o') if P 7 ^ p' 

P|[if (a) ei else 62] p cr = (ei, cr) if [a] cr = true 


prune E p x e ^ e 

prune{p, e) =p' where (p^ (t) = P|[e| p • 

(a) Pruning definitive writes. 


± □ dir, file(srr), dne □ T 

je] & a ^ a 
J[i£|CT = CT 

_[err]CT = (3- 

[mkdir(p)|(7 = <5-[p: = dir] 
|[creat(p, sir)|(7 = (5-[p: = file(sir)] 
|rm(p)|(7 = <5-[p: = dne] 

[if (a) ei else e2ld- = [ei|d- □ [e2ld- 
lei;e2ja= [e2l([ei]CT) 

(b) Detecting definitive writes. 


Figure 10; Shrinking resources. 


2. Instead of using an SMT solver, we tried to encode FS 
programs as binary-decision diagrams (BDDs) by ex¬ 
ploiting the natural hierarchy of paths to pick a good vari¬ 
able order, (e.g., a < a/b.) In our experience, the SMT 
solver was faster and significantly easier to use. For ex¬ 
ample, properties such as “all paths must be distinct” are 
very easy to express using distinct constraints in an 
SMT solver. 

5. Beyond Determinism 

After we’ve checked that a manifest is deterministic, we 
can treat it as an expression rather than a resource graph; 
we can pick any valid ordering of the resources (determin¬ 
ism ensures that they are all equivalent) and sequence them 
to form a single expression e. We emphasize that while re¬ 
source graphs denote relations, FS expressions denote func¬ 
tions. This lets Rehearsal check several properties quickly 
and easily. 

Invariants. We’ve seen that Puppet is actually very imper¬ 
ative. A manifest that declares a file resource may over¬ 
write it using some other resource, which is typically unde¬ 
sirable. Rehearsal checks for this issue using the following 
formula, which is unsatisfiable if e ensures that p is always 
a file with content str: 

3a.ok{e)a A f{e)d'{p) ^ file(sir) 

It is easy to imagine checks for several other invariants. 

Idempotence. We discussed in section |^that idempotence 
is a critical property of Puppet manifests. To test if a manifest 
is idempotent, we simply check if e = e; e holds. 

We emphasize that these checks are efficient because 
they do not have to consider all permutations of resources. 
Moreover, these simple checks would be unsound if applied 
to non-deterministic manifests. 


6. Evaluation 

Rehearsal is implemented in Scala and uses the Z3 Theorem 
Prover ED as its SMT solver. The majority of the codebase 
is the frontend that turns manifests into FS expressions. To 
model packages. Rehearsal needs to query an OS package 
manager. For portability, we’ve built a web service for Re¬ 
hearsal that can query the package manager for several op¬ 
erating systems. The service returns the package listing in 
a standardized format and stores the result in a database to 
speedup subsequent queries. Our current deployed service 
has Ubuntu and CentOS running in containers and it is easy 
to add support for other operating systems. 

Note that the times reported in this section do not in¬ 
clude the time required to fetch package listings. The pack¬ 
age querying tools (apt-cache and repoquery) can take 
several seconds to run, which is why our web service caches 
their results. 

Third-party benchmarks. We benchmark the determinism 
checker on a suite of 13 Puppet configurations gleaned from 
GitHub and Puppet Forge. We specifically chose bench¬ 
marks that did not use exec resources as detailed further 
in section We manually verified that six of them have 
determinism bugs and that seven do not. For each non- 
deterministic program, we developed a fix and verified that 
Rehearsal reports that it is deterministic and idempotent. We 
repeat all timing experiments ten times and report the av¬ 
erage. We perform all experiments on a quad-core 3.5 GHz 
Intel Core i5 with 8GB RAM. 

Figure [TT] shows the effect of pruning on Rehearsal’s de- 
terminacy analysis. In the figure, the non-deterministic man¬ 
ifests are marked -nondet. Without commutativity checking, 
four benchmarks do not complete in over ten minutes and 
one takes over two minutes to run (figure in^- Even with 
commutativity-checking, two benchmarks timeout after ten 
minutes. However, when commutativity-checking is coupled 
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(a) Paths per state. 


(b) Shrinking and eliminating resources. 


(c) Checking commutativity. 


Figure 11: Benchmarking determinacy analysis. 



Benchmark 


Figure 12: Benchmarking idempotence checking. 



This renders the commutativity-check useless, so Rehearsal 
is forced to explore all n! paths through the resource graph. 
Moreover, the file cannot be pruned either. Figure [T3] shows 
the running time grows non-linearly with n. In fact, even 
with n = 6, the running time exceeded two minutes. 

Although the simple benchmark described above can be 
constructed using FS, it is not a valid Puppet manifest, since 
Puppet does not allow multiple file-resources to affect the 
same path. A working alternative is to find n conflicting 
packages that all create the same file p and try to install all 
of them simultaneously. Even in this scenario. Rehearsal can 
determine that the manifest is non-deterministic relatively 
quickly {i.e., that the formula is satisfiable). However, we 
can force the manifest to be deterministic by using a single 
file resource that updates the contents of p after all the 
packages are installed: 

# All packages create a file /a 

package{^A-l’: before => FileEVa'] } 

package{'A- 2 ’: before => File[Va^] } 

package{'A- 3 ’: before => File[Va^] } 

file{Va^: content => } 


Figure 13: Scalability with n interfering resources. 


with file pruning (section [A^ , all benchmarks complete in 
less than two seconds (figure llb| i. Figure |1 la| shows the 
number of files in each manifest, with and without prun¬ 
ing. (Note that commutativity-checking does not affect the 
number of files.) As expected, the runtime of benchmarks 
corresponds to the number of files that need to be modeled. 

Figure [T^ reports the time required by the idempotence 
check is less than one second on all benchmarks. In practice, 
the idempotence check would be preceded by a determinism 
check, which typically takes more time to complete. 


Synthetic benchmarks. The benchmarks above suggest 
that commutativity checking and pruning are effective in 
practice. However, it is quite straightforward to construct 
an artificial scenario where the commutativity check is in¬ 
effective. The natural way to construct this benchmark is to 
have n unordered file-resources that write to the same path. 


The final file-resource makes the manifest deterministic, 
which forces the solver to construct a proof of unsatisfiabil¬ 
ity instead of terminating early with a satisfying assignment. 
We believe that this kind of scenario is very unlikely to arise 
in practice. 

Bugs found. Rehearsal found determinism bugs in six 
benchmarks (including a previously undiscovered bug). The 
bugs are of the kind we described in section]^ Specifically, 
several benchmarks omitted a necessary dependency be¬ 
tween a package and a configuration file. In addition, one 
benchmark omitted a dependency between a user account 
and SSH keys for the user. Broadly speaking, resource-types 
such as files and packages have a well-understood semantics, 
but users may not understand their interactions. 

7. Related Work 

Other system configuration languages. Several system 
configuration languages have been developed over decades 
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of research, many of which are surveyed by Delaet and 
Joosen ©. To the best of our knowledge, the kind of veri- 
hcation tools we have developed for Puppet have not been 
developed for these languages. Instead, we highlight how 
several languages differ from Puppet and consider what it 
would take to adapt our approach for them. 

Hagemark and Zadeck’s Site tool M has a DSL that is 
closely related to Puppet. A Sitehle describes bits of conhg- 
urations in “classes” that can be composed in several ways. 
Site traverses these classes in topological order and can also 
suffer missing dependencies, which our techniques detect. 

LCFG 11 provides built-in components for conhguring 
common applications. However, while new LCFG compo¬ 
nents have to be authored in Perl, Puppet encourages aver¬ 
age users to build their own abstractions using the Puppet 
DSL. An inter-component dependency in LCFG requires co¬ 
ordination between the conhguration hie and Perl code (us¬ 
ing “context variables”). Rehearsal leverages Puppet’s high- 
level DSL which makes all dependencies manifest. Building 
similar tools for LCFG would be difficult due to Perl. 

Anderson and Hetry H develop a denotational seman¬ 
tics for the SmartFrog conhguration language that faithfully 
models its non-deterministic semantics. They show that their 
model helps resolve several implementation issues, though 
ordering issues remain. They argue that system conhgura¬ 
tion languages need formal models and warn that popular 
languages gain features faster than formal models can be de¬ 
veloped. Our work shows that it is possible to model and 
analyze a signihcant fraction of a large system conhguration 
language, but we don’t disagree with their conclusions. 

Engage lUIl is a system for deploying and conhguring 
distributed applications that can specify complex, inter¬ 
machine dependencies, where values computed by one re¬ 
source at runtime can be used as inputs to another resource 
on a different machine. Puppet is more limited and does 
not support orchestration. To manage the life-cycle of a re¬ 
source, Engage users have to write drivers in Python. Al¬ 
though the Engage type-checker ensures that resources are 
composed correctly, it assumes that these Python drivers are 
error-free. In contrast, the Puppet DSL performs operations 
similar to Engage drivers and our tools can check this code. 

Nixos cni takes a radically different approach to pack¬ 
age and conhguration management than a typical Linux dis¬ 
tribution. NixOS places every package and conhguration in 
a unique location (determined during conhguration) and en¬ 
sures that they are immutable. This design forces NixOS 
policies to make all dependencies explicit. Puppet bring 
some of the advantages of NixOS to traditional operating 
systems and Linux distributions, but our paper shows that it 
doesn’t provide the same guarantees of NixOS. Instead of 
proposing a radical, new architecture, we show that program 
verihcation techniques can be employed to provide strong 
guarantees for Puppet conhgurations. 


Tucker and Krishnamurthi ll29l argue that Racket’s unit 
system could be adapted to build a better package manager. 
The benehts of their design are similar to the benehts of 
NixOS (discussed above). 

Testing and verification of configurations. CloudMake 
is a cloud-based build system in use at Microsoft that has 
important features such as artifact caching, parallel builds, 
etc. CloudMake commands make all inputs and outputs 
explicit. Christakis, et al. JS) have a mechanized proof that 
CloudMake scripts are race-free, which justihes parallel 
builds. Our paper shows that it’s not possible to prove such 
a theorem for all Puppet conhgurations. Instead, Rehearsal 
verihes that individual manifests are deterministic. 

Hummer et al. systematically test Chef conhgura¬ 
tions and hnd that several conhgurations are not idempotent. 
Their test-based approach cannot ensure complete coverage 
and can take several days. By contrast, we use static analy¬ 
sis to prove determinacy and idempotency, which would be 
more difficult to do for Chef as it is a Ruby-embedded DSL. 

Although Puppet uses native package managers to imple¬ 
ment package resources. Puppet doesn’t leverage the rich 
information that packages provide, such as their direct de¬ 
pendencies and conhicts, which leads to the kind of errors 
described in section It should be possible to leverage 
package metadata to build more useful verihcation tools, 
perhaps using the SAT-based encoding of Opium ll2^ . Un¬ 
like apt-get. Opium’s algorithm for calculating installa¬ 
tion/uninstallation is complete for a given distribution. The 
analogous problem for Puppet would be to calculate the 
installation prohle for a resource, given a universe of re¬ 
sources, such as modules on Puppet Eorge. To do so, one 
would need to calculate and verify dependencies. Rehearsal 
does the latter and could be augmented to do the former. 

Rehearsal uses a straightforward model of the hlesystem, 
partly because Puppet’s model hides many platform-specihc 
hlesystem details for portability (e.g.. Puppet doesn’t sup¬ 
port hard links). Others have developed hlesystem models 
that are much richer than ours (e.g., P1l20ll22ll '). The program 
logic of Gardner et al. m is particularly interesting because 
it enables modular reasoning about hlesystem-manipulating 
programs. In contrast, the verihcation techniques in our pa¬ 
per are not modular because we support Puppet features that 
have global effects on the resource graph. If these features 
were ignored, a modular analysis would be attractive. 

Cloud services such as Microsoft Azure contain large 
conhgurations with many components in various represen¬ 
tations (e.g.YAML, XML, INI, etc.). ConfValley ifTSll uni- 
hes these conhgurations into a single representation and val¬ 
idates them with respect to user-written predicates about the 
conhguration. The predicates may describe desirable prop¬ 
erties for a particular cloud service conhguration such as en¬ 
suring that a particular variable has the correct type or a cer¬ 
tain hie has the appropriate permissions. Rehearsal verihes 
two specihc properties about the effects of a Puppet conhg- 
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uration on a machine, rather than properties of the conhg- 
uration itself. We consider every possible input and execu¬ 
tion path in order to prove or disprove idempotence and de¬ 
terminism. A ConfValley-style verihcation of Puppet would 
involve writing predicates about the structure of the resource 
graph, which should be straightforward to do with our tools. 

Determinacy checkers. In the past few years, several tools 
have been developed that use static ini ED H] and dy¬ 
namic EEll techniques to check that multi-threaded pro¬ 
grams are deterministic. Rehearsal is a static determinacy 
checker for Puppet and leverages an SMT solver, thus is 
most closely related to Liquid Effects ifTTll . Liquid Effects 
establishes determinism by showing that concurrent effects 
are disjoint, but there are common examples of deterministic 
Puppet programs that do not have disjoint effects. Instead, 
Rehearsal has a commutativity check that accounts a pat¬ 
tern of false sharing that is common to Puppet (section [43] ). 
Rehearsal and Liquid Effects address determinism in two 
very different domains. Liquid Effects proves determinism 
for multi-threaded C programs with pointers, aliasing, and 
functions that are tackled in a modular way with types. In 
contrast. Puppet manifests have no aliasing, loops, or proce¬ 
dures. Since our problem is simpler, we are able to build a 
scalable, sound, and complete determinacy checker that re¬ 
quires no annotations by the programmer. 

When Rehearsal reports that a Puppet manifest is deter¬ 
ministic, the manifest may still yield different outputs for 
different inputs, i.e.. Rehearsal only verihes that a manifest 
maps each input state to a single output state. In contrast, 
Andreasen and Mpller 13), have developed techniques to in¬ 
fer that program expressions determinate, i.e., that an ex¬ 
pression produces the same value in all executions. They 
exploit determinate expressions to improve the precision of 
their JavaScript Type Analyzer. In contrast. Puppet expres¬ 
sions are always determinate, but Puppet manifests can be 
non-deterministic. 

Alternate uses of configuration management. Einally, we 
note that configuration management is an overloaded term 
in the literature. This paper addresses an issue that arises in 
software conhguration and deployment. However, the term 
conhguration management is also used to refer to version- 
control systems {e.g., CVS and Git) and to application con¬ 
hguration CSIIIT), which is not the subject of this work. 

8. Limitations 

The primary limitation of this work is that Puppet manifests 
support embedded shell scripts (using the exec resource 
type). Shell scripts are often an anti-pattern, but they are 
indispensable for certain tasks. Eor example, they are often 
used to setup software that has not made its way into sanc¬ 
tioned software repositories. The main challenge with shell 
scripts is that they can have arbitrary effects on the hlesys- 
tem, unlike the other resource-types that have a clearer se¬ 
mantics and lend themselves to formal models. 


Another limitation of our work is that our analyses rely 
on models of system resources, which can be inaccurate. 
For example, to model packages, we need to know the hies 
that a package creates. At present, we assume that pack¬ 
ages only create the hies returned by apt-file (on Debian) 
and repoquery (on Red Hat). However, many packages use 
“post-install scripts” to create additional hies, which our ap¬ 
proach will miss. Therefore, although our algorithms are 
sound and complete with respect to our model of system re¬ 
sources, our models have known limitations. A more precise 
alternative would be to actually install packages in a sand¬ 
boxed environment and check what hies get written to disk. 

Finally, as suggested above, our analysis is platform- 
dependent. In fact, the choice of operating system deter¬ 
mines how packages are modeled. Although Puppet has sev¬ 
eral platform-neutral features, it also exposes the platform 
name and version as program variables that a manifest can 
use to specialize for a particular platform. Rehearsal takes 
the platform name as a command-line hag and so a manifest 
can be re-verihed for several platforms. However, it would 
be more useful to check that a manifest has similar effects 
on different platforms. 

9. Conclusion 

This paper presents Rehearsal, the hrst verihcation tool for 
Puppet, a popular system conhguration language. Specih- 
cally. Rehearsal checks that exec-free Puppet manifests are 
deterministic and idempotent, which are both fundamental 
properties of correct Puppet manifests. To build Rehearsal, 
we developed a simple semantics for Puppet that we hope 
will be useful to other researchers. We believe that our ap¬ 
proach to modeling Puppet will enable several other tools, 
e.g., manifest repair and synthesis, and security auditing. 
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